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ASSISTANT  SECRETARY  OF  THE  NAVY  (FINANCIAL 
MANAGEMENT  AND  COMPTROLLER) 

DIRECTOR,  ADVANCED  RESEARCH  PROJECTS 
AGENCY 

SUBJECT:  Evaluation  of  Advanced  Research  Projects  Agency  Source-Selection  of 
Hydrothermal  Oxidation  of  Navy  Shipboard  Excess  Hazardous  Material 
(Project  No.  5PT-8012) 


Introduction 

We  are  providing  this  report  for  your  information  and  use.  The  evaluation  is  in 
response  to  a  Defense  Hotline  complaint  concerning  the  source-selection 
procedures  for  the  Advanced  Research  Projects  Agency  (ARP A)  Broad  Agency 
Announcement  (BAA)  94-45  for  "Hydrothermal  Oxidation  of  Navy  Excess 
Hazardous  Material. " 

BAA-94-45  was  issued  for  research,  development,  design,  fabrication,  and 
shoreside  testing  of  a  full-scale  hydrothermal  oxidation  reactor  module  that  had 
the  capacity  to  destroy  100  pounds  per  hour  of  Navy  shipboard  excess 
hazardous  materials.  Discharge  from  the  reactor  would  be  nontoxic  to  humans 
and  the  environment  and  would  meet  appropriate  international.  Federal,  state, 
and  local  environmental  regulations.  ARP  A  planned  to  fund  as  many  proposals 
as  met  the  BAA  objectives  and  fell  within  the  $20  million  allocated  for  the 
program. 

The  complainant  alleged  that: 

Evaluation  of  the  BAA  proposals  was  conducted  in  a  very  short  period, 
allowing  insufficient  time  for  a  fair  evaluation. 

Information  concerning  the  technical  aspects  of  the  BAA  was  not 
distributed  equitably  to  the  potential  bidders,  penalizing  the  complainant. 

Notice  of  non-selection  was  sent  to  a  remotely  located  subcontractor  of 
the  complainant,  limiting  the  time  available  to  file  a  formal  protest. 


ARPA  misled  the  complainant  and  held  other  bidders  to  a  different 
standard.  No  clear  objectives  or  consistently  applied  criteria  were  used  to 
evaluate  the  merits  of  diverse  proposals. 

ARPA  inappropriately  issued  a  BAA  instead  of  a  Request  for  Proposal 
and  established  unrealistic  and  incompatible  goals  for  the  BAA,  either 
knowingly  or  through  incompetence. 


Evaluation  Results 

We  reviewed  the  complainant's  allegations  contained  in  his  letter  and  in  his 
interview  with  us.  We  did  not  substantiate  the  allegations.  Enclosure  1 
discusses  the  allegations. 


Evaluation  Objectives 

Our  objective  for  this  evaluation  was  to  evaluate  the  source-selection  procedures 
for  BAA-94-45  and  to  verify  the  allegations  about  the  ARPA  procurement 
process  with  respect  to  the  BAA. 


Scope  and  Methodology 

The  scope  of  the  evaluation  included  a  review  of  the  state  of  development  of 
hydrothermal  oxidation  technology.  The  evaluation  began  on 
February  22,  1995  and  was  completed  on  July  14,  1995.  We  reviewed  program 
requirements,  selection  process  documentation,  evaluation  team  members' 
qualifications,  and  negotiations  conducted  for  the  winning  proposals.  We  did  a 
detailed  review  of  the  three  winning  proposals  and  the  complainant's  proposal, 
independently  assessed  the  winning  and  losing  proposals'  technical  merit,  and 
.  assessed  the  panel's  application  of  source-selection  criteria  to  proposal 
evaluation. 

We  reviewed  data  from  the  source-selection  process,  starting  with  the  BAA 
announcement  and  continuing  through  source  selection.  For  background,  we 
reviewed  the  role  of  Office  of  Naval  Research  participation  in  establishing  the 
goals  of  the  BAA  and  the  relationship  between  ARPA  project  management  and 
its  Office  of  Naval  Research  contracting  agent.  We  also  reviewed  the  status  of 
a  winning  bidder's  existing  contract  for  a  similar  system.  The  dates  of  the 
documentation  reviewed  covered  the  period  of  June  1991  to  April  1995. 


2 


Background 

The  Navy  has  been  involved  with  the  disposal  of  shipboard-generated  hazardous 
waste  for  several  years  and  has  been  considering  the  hydrothermal  oxidation 
disposition  method.  The  Navy  is  also  concerned  about  the  treatment  of  gray 
and  black  water,  due  to  increasingly  strict  disposal  regulations.  The  Office  of 
Naval  Research,  with  other  Navy  organizations,  provided  its  requirements  for 
hazardous  waste  disposal  to  ARPA. 

Goals  for  the  BAA  were  generally  very  demanding  regarding  size,  weight,  and 
throughput  of  waste  to  be  processed.  Total  dimensions  of  the  module 
containing  the  unit,  including  the  reactor,  all  pumps,  tankage,  and  controls, 
were  to  be  less  than  8  feet  by  9  feet  by  10  feet,  approximately  half  the  size  of  a 
standard  shipping  container.  Weight  was  limited  to  a  goal  of  8,000  pounds  to 
ensure  transportability  to  and  from  the  deck  of  a  ship  by  helicopter  or  crane. 
The  entire  unit  was  to  be  self-contained  with  only  electrical  and  plumbing 
connections  to  the  ship.  Input  to  the  reactor  was  defined  as  shipboard-generated 
hazardous  materials;  however,  potential  bidders  were  also  to  consider  gray  and 
black  water  processing.  Operators  would  deliver  the  waste  to  the  reactor  in  55- 
gallon  steel  drums.  Water  and  solid  output  from  the  hydrothermal  oxidation 
reactor  was  to  depart  in  a  straightforward  manner  that  required  little 
management  and  no  specially  trained  personnel.  Processing  volume  of  the 
system  was  set  at  1,000  pounds  per  day  and  operation  was  limited  to  a 
maximum  of  10  hours  per  day.  The  BAA  required  sufficient  automation  of 
system  controls  to  ensure  a  capability  of  operation  by  shipboard  personnel  with 
little  training  and  workload. 


Discussion 

We  did  not  substantiate  the  allegations.  Even  though  the  source  selection  was 
conducted  in  a  short  time,  the  evaluations  were  fair.  The  same  basic 
information  was  provided  to  all  potential  bidders  and  answers  to  bidders' 
questions  were  consistent,  giving  none  an  advantage.  Selection  criteria  were 
established  before  the  evaluation  and  all  bidders  were  held  to  the  same  standard. 
We  also  determined  that  because  of  the  demanding  goals  sought  for  this 
application,  a  BAA  was  more  appropriate  than  a  more  definitive  specification. 

Our  evaluation  identified  the  lack  of  fully  documented  source-selection 
worksheets.  Subsequent  interviews  with  the  source-selection  panel  members 
found  that  they  had  sufficiently  considered  the  relevant  criteria.  The  source- 
selection  process  can  be  a  subject  of  criticism  if  the  documentation  is  not  very 
detailed. 
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Management  Comments 

We  provided  a  draft  of  this  report  on  August  16,  1995.  Because  this  report 
contains  no  findings  or  recommendations,  comments  were  not  required,  and 
none  were  received.  Therefore,  we  are  publishing  this  report  in  final  form. 

We  appreciate  the  courtesies  extended  to  the  staff.  A  list  of  the  evaluation  team 
members  is  inside  the  back  cover  of  this  report.  Enclosure  3  lists  the 
distribution  of  this  report. 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 


Enclosures 
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Allegations  and  Evaluation  Comments 


Our  evaluation  concluded  that  no  evidence  corroborated  the  allegations.  We 
considered  the  allegations  contained  in  the  letter  to  the  Deputy  Inspector 
General,  DoD,  and  others  in  our  subsequent  interview  with  the  complainant. 
The  allegations  and  our  evaluation  of  their  validity  are  summarized  below. 

Allegation  1.  Evaluation  of  the  BAA  proposals  was  conducted  in  a  very 
short  period,  allowing  insufficient  time  for  a  fair  evaluation. 

Evaluation  Comments.  The  Government  review  panel  for  BAA-94-45 
received  the  proposals  Saturday,  November  5,  1994,  for  reading  before  the 
scheduled  November  8  through  10,  1994,  review  panel  meeting.  The  panel  was 
given  the  evaluation  guidelines  and  a  copy  of  the  proposal  evaluation  worksheet. 
The  panel  was  asked  to  review  as  much  of  the  material  as  possible  over  the 
weekend.  They  met  November  8,  1994,  for  the  three  scheduled  days  of  source 
selection. 

All  panel  members  had  read  most  of  the  12  proposals  before  the  start  of  the 
review  meeting.  By  the  conclusion  of  the  meeting,  all  panelists  had  read  and 
scored  the  three  winning  proposals.  Each  losing  proposal  had  been  read  by  at 
least  five  panel  members.  Each  losing  proposal  had  been  scored  by  at  least 
three  panel  members. 

ARPA  explained  that  the  quick  turn-around  time  for  the  evaluation  was  an 
attempt  to  expedite  the  contract  award  process.  The  time  spent  and  the 
procedure  the  panel  used  were  sufficient  for  a  fair  evaluation. 

Allegation  2.  Information  concerning  the  technical  aspects  of  the  BAA  was 
not  distributed  equitably  to  the  potential  bidders,  penalizing  the 
complainant. 

Evaluation  Comments.  ARPA  provided  all  interested  parties  identical 
information  in  "Hydrothermal  Oxidation  Proposer  Information  Pamphlet. "  The 
package  consisted  of  the  Broad  Agency  Announcement;  Proposer  Information, 
which  provides  detailed  information  on  the  proposal  format;  a  copy  of  the 
required  cover  sheet;  a  copy  of  Standard  Form  1411,  "Contract  Pricing 
Proposal  Cover  Sheet;"  and  a  pie  chart  showing  the  relative  quantities  of  several 
different  types  of  excess  hazardous  materials  found  aboard  a  typical  aircraft 
carrier. 

The  ARPA  project  manager  answered  questions  asked  by  several  proposers.  All 
bidders  were  treated  equitably.  We  could  not  prove  the  allegation  that  the 
complainant  received  any  less  or  more  information  that  would  have  placed  him 
at  a  disadvantage  with  respect  to  the  proposal  evaluation. 
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Allegations  and  Evaluation  Comments 


Allegation  3.  Notice  of  non-selection  was  sent  to  a  remotely  located 
subcontractor  of  the  complainant,  limiting  the  time  available  to  file  a 
formal  protest. 

Evaluation  Comments.  While  the  notice  of  non-selection  was  sent  to  a 
subcontractor  of  the  complainant,  Federal  Acquisition  Regulation  33.103(2) 
states: 


Protests,  based  on  alleged  improprieties  in  a  solicitation,  which  are 
apparent  prior  to  bid  opening  or  the  closing  date  for  receipt  of  proposals 
shall  be  filed  prior  to  bid  opening  or  the  closing  date  for  receipt  of 
proposals.  In  all  other  cases,  protests  shall  be  filed  not  later  than  10 
working  days  after  the  basis  of  protest  is  known  or  should  have  been 
known,  whichever  is  earlier.  The  agency  for  good  cause  shown,  or  where 
it  determines  that  a  protest  raises  issues  significant  to  the  agency’s 
acquisition  system,  may  consider  any  protest  which  is  not  filed  timely 
(i emphasis  is  added). 

Under  the  above  Federal  Acquisition  Regulation  clause,  the  complainant  had  10 
working  days  to  file  a  protest  after  becoming  aware  of  the  perceived  problem 
and  was  not  hindered  by  inappropriate  routing  of  the  letter  of  notice. 

Allegation  4.  ARPA  misled  the  complainant  and  held  other  bidders  to  a 
different  standard.  No  clear  objectives  or  consistently  applied  criteria  were 
used  to  evaluate  the  merits  of  diverse  proposals. 

Evaluation  Comments.  The  clarifications  that  were  requested  by  potential 
proposers,  including  the  complainant,  before  proposal  submittal  were  reviewed. 
All  responses  to  clarifications  were  consistent  with  the  published  BAA.  In 
essence,  several  different  potential  offerers  requested  the  same  clarifications. 
ARPA  responded  the  same  to  each  clarification.  ARPA  did  not  mislead  the 
complainant  or  any  other  potential  offerer. 

The  Government  review  panel  was  given  sufficient  documentation  to  help  it 
impartially  evaluate  the  proposals.  BAA-94-45  provided  the  basis  for  the 
evaluation.  BAA-94-45  "Evaluation  Guidelines"  explained  the  mission  and 
goals  of  the  program.  It  contained  evaluation  criteria  for  each  of  the  four  areas 
being  considered.  The  BAA-94-45  "Proposal  Evaluation  Worksheet"  gave  a 
framework  to  enable  the  evaluators  to  consistently  rate  each  proposal.  There 
was  no  evidence  that  ARPA  held  the  offerers  to  different  standards.  The 
information  provided  to  the  offerers  stated  clear  goals.  The  ARPA  source- 
selection  panel  applied  appropriate  evaluation  criteria  that  permitted  the  fair 
evaluation  of  proposals  containing  diverse  technical  approaches. 
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Allegations  and  Evaluation  Comments 


Allegation  5.  ARPA  inappropriately  issued  a  BAA  instead  of  a  Request  for 
Proposal  and  established  unrealistic  and  incompatible  goals  for  the  BAA, 
either  knowingly  or  through  incompetence. 

Evaluation  Comments.  Because  of  the  nature  of  the  work  being  done  by 
ARPA,  most  of  its  solicitations  take  the  form  of  a  BAA  in  accordance  with 
Federal  Acquisition  Regulation  6.102(d)(2).  ARPA  has  developed  internal 
BAA  policies  and  procedures  based  on  Federal  Acquisition  Regulation  35.016. 
As  part  of  general  ARPA  procurement  process,  a  decision  is  made  on  the  type 
of  procurement  vehicle  to  use,  based  on  the  nature  of  the  research  being 
considered. 

A  Request  for  Proposals  could  have  been  more  appropriate  for  this  solicitation 
considering  the  nature  of  the  shipboard  system  ARPA  desired.  The  purpose  of 
the  system,  hazardous  waste  disposal  aboard  a  ship,  made  it  necessary  to  impose 
limitations  dictated  by  the  shipboard  environment.  Those  limitations  could 
cause  the  goals  of  the  BAA  to  be  more  appropriate  than  procurement  under  a 
Request  for  Proposals. 

Following  the  general  ARPA  procurement  process  and  specific  ARPA  BAA 
policy,  ARPA  decided  that  a  BAA  was  the  appropriate  vehicle  for  the 
hydrothermal  oxidation  research.  The  research  goal  was  to  extend  the  state  of 
the  art  and  to  learn  the  present  limits  for  the  desired  application.  All  proposers 
faced  the  same  uncertainty  and  were  technically  challenged  to  provide  a  solution 
for  the  solicitation.  We  consider  use  of  a  BAA  appropriate  for  this  award. 
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Organizations  Visited  or  Contacted 


Department  of  the  Navy 

Office  of  Naval  Research,  Arlington,  VA 

Carderock  Division-Naval  Surface  Warfare  Center,  Annapolis,  MD 


Other  Defense  Organization 
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Enclosure  2 


Report  Distribution 


Office  of  the  Secretary  of  Defense 
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Deputy  Comptroller  (Program/Budget) 

Assistant  to  the  Secretary  of  Defense  (Public  Affairs) 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Navy 
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Non-Defense  Federal  Organizations  and  Individuals  (Continued) 

Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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